<!DOCTYPE html>


<html lang="zh-CN">


<head>
  <meta charset="utf-8" />
  <meta name="baidu-site-verification" content="code-kg5UjKJZM2" />
   
  <meta name="keywords" content="活,炼" />
   
  <meta name="description" content="shimmerjordan" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
  <title>
    Python爬虫入门 |  丛烨-shimmerjordan
  </title>
  <meta name="generator" content="hexo-theme-ayer">
  
  <link rel="shortcut icon" href="/favicon.ico" />
  
  
<link rel="stylesheet" href="/dist/main.css">

  <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/Shen-Yu/cdn/css/remixicon.min.css">
  
<link rel="stylesheet" href="/css/custom.css">

  
  <script src="https://cdn.jsdelivr.net/npm/pace-js@1.0.2/pace.min.js"></script>
  
  

<script type="text/javascript">
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');

ga('create', 'G-Q0DT8B8VJW', 'auto');
ga('send', 'pageview');

</script>



  
<script>
var _hmt = _hmt || [];
(function() {
	var hm = document.createElement("script");
	hm.src = "https://hm.baidu.com/hm.js?6d06f826e125297d4ce0fa7a1449328e";
	var s = document.getElementsByTagName("script")[0]; 
	s.parentNode.insertBefore(hm, s);
})();
</script>


<link rel="alternate" href="/atom.xml" title="丛烨-shimmerjordan" type="application/atom+xml">
</head>

</html>

	<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/font-awesome/css/font-awesome.min.css">
	<script src="https://cdn.jsdelivr.net/gh/stevenjoezhang/live2d-widget@latest/autoload.js"></script>


<body>
  <div id="app">
    
      
    <main class="content on">
      <section class="outer">
  <article
  id="post-Python爬虫入门"
  class="article article-type-post"
  itemscope
  itemprop="blogPost"
  data-scroll-reveal
>
  <div class="article-inner">
    
    <header class="article-header">
       
<h1 class="article-title sea-center" style="border-left:0" itemprop="name">
  Python爬虫入门
</h1>
 

    </header>
     
    <div class="article-meta">
      <a href="/2020/02/02/Python-crawlers-beginning/" class="article-date">
  <time datetime="2020-02-02T12:41:59.000Z" itemprop="datePublished">2020-02-02</time>
</a> 
  <div class="article-category">
    <a class="article-category-link" href="/categories/Python%E5%88%9D%E7%BA%A7%E5%BA%94%E7%94%A8/">Python初级应用</a>
  </div>
  
<div class="word_count">
    <span class="post-time">
        <span class="post-meta-item-icon">
            <i class="ri-quill-pen-line"></i>
            <span class="post-meta-item-text"> 字数统计:</span>
            <span class="post-count">4.8k</span>
        </span>
    </span>

    <span class="post-time">
        &nbsp; | &nbsp;
        <span class="post-meta-item-icon">
            <i class="ri-book-open-line"></i>
            <span class="post-meta-item-text"> 阅读时长≈</span>
            <span class="post-count">22 分钟</span>
        </span>
    </span>
</div>
 
    </div>
      
    <div class="tocbot"></div>




  
    <div class="article-entry" itemprop="articleBody">
       
  <h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>在初步掌握Python基本语法后，这里使用Py进行一个简单的爬取图片应用作为入门训练。<br>首先爬虫的基本步骤可以概括为：</p>
<ul>
<li>输入域名 -&gt; 下载源代码 -&gt; 分析图片路径 -&gt; 下载图片</li>
</ul>
<p>这里我们选取<a target="_blank" rel="noopener" href="http://www.meizitu.com/a/pure.html">http://www.meizitu.com/a/pure.html</a>作为爬取的目标网站。因为这个网站好分析，爬取简单。</p>
<span id="more"></span>
<h1 id="分析页面"><a href="#分析页面" class="headerlink" title="分析页面"></a>分析页面</h1><p><img src="https://s2.ax1x.com/2020/02/02/1tR0mD.jpg" alt="1tR0mD.jpg"></p>
<p>爬虫最重要的一点，便是寻找到分页地点。因为分页处代表可能存在规律可循。建立在既有规律的基础上，我们可以更智能、准确、方便得爬取到相关数据</p>
<p>以本网站为例，我们可从上述图片中发现分页：<br><img src="https://s2.ax1x.com/2020/02/02/1tTWwV.jpg" alt="1tTWwV.jpg"></p>
<p>如图所示，使用浏览器的开发者工具可发现分页规律：</p>
<pre><code>http://www.meizitu.com/a/pure_1.html
http://www.meizitu.com/a/pure_2.html
http://www.meizitu.com/a/pure_3.html
http://www.meizitu.com/a/pure_4.html
</code></pre><p>接下来我们使用Python实现这部分（采用面向对象的思想）</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line">all_urls = []   <span class="comment">#我们拼接好的图片集和列表路径</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Spider</span>():</span><br><span class="line">    <span class="comment">#构造函数，初始化数据使用</span></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self, target_url, headers</span>):</span><br><span class="line">        self.target_url =  target_url</span><br><span class="line">        self.headers = headers</span><br><span class="line"></span><br><span class="line">    <span class="comment">#获取所有的想要爬取的URL</span></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">getUrls</span>(<span class="params">self, start_page, page_num</span>):</span><br><span class="line">        <span class="keyword">global</span> all_urls</span><br><span class="line">        <span class="comment">#循环得到URL</span></span><br><span class="line">        <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(start_page, page_num + <span class="number">1</span>):</span><br><span class="line">            url = self.target_url % i   <span class="comment">#注意这里并非取余操作，而是格式化字符串</span></span><br><span class="line">            all_urls.append(url)</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">&quot;__main__&quot;</span>:</span><br><span class="line">    headers = &#123;</span><br><span class="line">        <span class="comment"># 这部分在浏览器中输入 about:version查看</span></span><br><span class="line">        <span class="string">&#x27;User-Agent&#x27;</span>: <span class="string">&#x27;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.62 Safari/537.36&#x27;</span>,</span><br><span class="line">        <span class="string">&#x27;HOST&#x27;</span>: <span class="string">&#x27;www.meizitu.com&#x27;</span></span><br><span class="line">    &#125;</span><br><span class="line">    target_url = <span class="string">&#x27;http://www.meizitu.com/a/pure_%d.html&#x27;</span>    <span class="comment">#图片集和列表规则</span></span><br><span class="line"></span><br><span class="line">    spider = Spider(target_url, headers)</span><br><span class="line">    spider.getUrls(<span class="number">1</span>,<span class="number">16</span>)</span><br><span class="line">    <span class="built_in">print</span>(all_urls)</span><br></pre></td></tr></table></figure>
<p>以上的代码主要有以下几个要点：</p>
<ul>
<li>声明了一个<code>class Spider()</code>类，并使用<code>def __init__</code>声明一个构造函数  </li>
<li>拼接URL，我们可以选择很多方法，这里采用最简单直接的字符串拼接方法  </li>
<li>注意上述代码中有一个全局变量<code>all_urls</code>，用来存储我们的所有分页的URL  </li>
</ul>
<h1 id="爬虫核心"><a href="#爬虫核心" class="headerlink" title="爬虫核心"></a>爬虫核心</h1><p>我们需要分析页面中的逻辑。首先打开<a target="_blank" rel="noopener" href="http://www.meizitu.com/a/pure_1.html">http://www.meizitu.com/a/pure_1.html</a>，右键审查但愿元素：<br><img src="https://s2.ax1x.com/2020/02/03/1UCMVS.jpg" alt="1UCMVS.jpg"><br><img src="https://s2.ax1x.com/2020/02/03/1UiY7T.jpg" alt="1UiY7T.jpg"><br>上图红色框内的链接即为所需要的链接<br>点击图片后，可发现进入图片详情页面，发现尽然是套图，那么现在的问题在于：<br>首先，我们需要在 <a target="_blank" rel="noopener" href="http://www.meizitu.com/a/pure_1.html">http://www.meizitu.com/a/pure_1.html</a> 这种页面中爬取所有的 <a target="_blank" rel="noopener" href="http://www.meizitu.com/a/5585.html">http://www.meizitu.com/a/5585.html</a> 这种地址<br>这里我们采用多线程的方案进行爬取，此外还涉及到名为“观察者模式”的设计模式。<br>首先引入三个模块，分别为多线程、正则表达式、时间模块。并新增一个全局的变量，由于是多线程操作，这里需要引入线程锁：</p>
<pre><code>import threading    #多线程模块
import re            #正则表达式模块
import time            #时间模块

all_img_urls = []    #图片列表页面的数组
g_lock = threading.Lock()    #初始化一个锁
</code></pre><p>下面声明一个生产者的类，用来不断获取详情页地址，然后添加到<code>all_img_urls</code>这个全局变量中</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#生产者，负责从每个页面提取图片列表链接</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Producer</span>(threading.Thread):   </span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">run</span>(<span class="params">self</span>):</span><br><span class="line">        headers = &#123;</span><br><span class="line">            <span class="comment"># 这部分在浏览器中输入 about:version查看</span></span><br><span class="line">            <span class="string">&#x27;User-Agent&#x27;</span>: <span class="string">&#x27;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.62 Safari/537.36&#x27;</span>,</span><br><span class="line">            <span class="string">&#x27;HOST&#x27;</span>: <span class="string">&#x27;www.meizitu.com&#x27;</span></span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">global</span> all_urls</span><br><span class="line">        <span class="keyword">while</span> <span class="built_in">len</span>(all_urls) &gt; <span class="number">0</span> :</span><br><span class="line">            g_lock.acquire()  <span class="comment">#在访问all_urls的时候，需要使用锁机制</span></span><br><span class="line">            page_url = all_urls.pop()   <span class="comment">#通过pop方法移除最后一个元素，并且返回该值</span></span><br><span class="line">            </span><br><span class="line">            g_lock.release() <span class="comment">#使用完成之后及时把锁给释放，方便其他线程使用</span></span><br><span class="line">            <span class="keyword">try</span>:</span><br><span class="line">                <span class="built_in">print</span>(<span class="string">&quot;分析&quot;</span>+page_url)   </span><br><span class="line">                response = requests.get(page_url , headers = headers,timeout=<span class="number">3</span>)</span><br><span class="line">                all_pic_link = re.findall(<span class="string">&#x27;&lt;a target=\&#x27;_blank\&#x27; href=&quot;(.*?)&quot;&gt;&#x27;</span>,response.text,re.S)   </span><br><span class="line">                <span class="keyword">global</span> all_img_urls</span><br><span class="line">                g_lock.acquire()   <span class="comment">#这里还有一个锁</span></span><br><span class="line">                all_img_urls += all_pic_link   <span class="comment">#这个地方注意数组的拼接，没有用append直接用的+=也算是python的一个新语法吧</span></span><br><span class="line">                <span class="built_in">print</span>(all_img_urls)</span><br><span class="line">                g_lock.release()   <span class="comment">#释放锁</span></span><br><span class="line">                time.sleep(<span class="number">0.5</span>)</span><br><span class="line">            <span class="keyword">except</span>:</span><br><span class="line">                <span class="keyword">pass</span></span><br></pre></td></tr></table></figure>
<ul>
<li>上述代码从threading.Thread中继承了一个子类  </li>
<li>线程锁：在上述代码中，当我们操作<code>all_urls.pop()</code>时，我们不希望其他线程对其进行同时操作，否则可能发生意外。所以我们使用<code>g_lock.acquire()</code>锁定资源，然后使用完成之后，记住一定要立即释放<code>g_lock.release()</code>，否则此资源被一直占用，程序无法继续进行。  </li>
<li>匹配网页中的URL，这里使用正则表达式，后面我们会使用其它方法进行匹配。<br><code>re.findall()</code>方法是获取所有匹配到的内容  </li>
<li>这里使用<code>try: except</code>包装可能出现异常的代码</li>
</ul>
<p>如果上述代码均没有出现问题，那么我们可以在程序的入口处编写以下代码来执行程序</p>
<pre><code>for x in range(2):
    t = Producer()
    t.start()
</code></pre><p>因为我们的Producer继承自threading.Thread类。所以我们必须实现的一个方法是<code>def run</code>（已在上述的代码中呈现）。运行结果如下图所示：<br><img src="https://s2.ax1x.com/2020/02/04/1B0XvV.jpg" alt="1B0XvV.jpg"><br>接下来，我们需要执行这样一步操作，我想要等待图片详情页面全部获取完毕，在进行接下来的分析操作。<br>这里增加代码：</p>
<pre><code>#threads= []   
#开启两个线程去访问
for x in range(2):
t = Producer()
t.start()
#threads.append(t)

# for tt in threads:
#     tt.join()

print(&quot;进行到我这里了&quot;)
</code></pre><p>注释关键代码，运行如下：</p>
<pre><code>[linuxboy@bogon demo]$ python3 down.py
分析http://www.meizitu.com/a/pure_2.html
分析http://www.meizitu.com/a/pure_1.html
进行到我这里了
[&#39;http://www.meizitu.com/a/5585.html&#39;,
</code></pre><p>把上面的tt.join等代码注释打开，运行如下：</p>
<pre><code>[linuxboy@bogon demo]$ python3 down.py
分析http://www.meizitu.com/a/pure_2.html
分析http://www.meizitu.com/a/pure_1.html
[&#39;http://www.meizitu.com/a/5429.html&#39;, ......
进行到我这里了
</code></pre><p>发现一个本质的区别，就是，我们由于是多线程的程序，所以，当程序跑起来之后，<code>print(&quot;进行到我这里了&quot;)</code>不会等到其他线程结束，就会运行到，但是当我们改造成上面的代码之后，也就是加入了关键的代码 <code>tt.join()</code> 那么主线程的代码会等到所以子线程运行完毕之后，在接着向下运行。这就满足了，我刚才说的，先获取到所有的图片详情页面的集合，这一条件了。</p>
<p>join所完成的工作就是线程同步，即主线程遇到join之后进入阻塞状态，一直等待其他的子线程执行结束之后，主线程在继续执行。这个大家在以后可能经常会碰到。</p>
<p>下面编写一个消费者/观察者，也就是不断关注刚才我们获取的那些图片详情页面的数组。</p>
<p>添加一个全局变量，用来存储获取到的图片链接<br><code>pic_links = []            #图片地址列表</code></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#消费者</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Consumer</span>(threading.Thread) : </span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">run</span>(<span class="params">self</span>):</span><br><span class="line">        headers = &#123;</span><br><span class="line">            <span class="string">&#x27;User-Agent&#x27;</span>: <span class="string">&#x27;Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0&#x27;</span>,</span><br><span class="line">            <span class="string">&#x27;HOST&#x27;</span>:<span class="string">&#x27;www.meizitu.com&#x27;</span></span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">global</span> all_img_urls   <span class="comment">#调用全局的图片详情页面的数组</span></span><br><span class="line">        <span class="built_in">print</span>(<span class="string">&quot;%s is running &quot;</span> % threading.current_thread)</span><br><span class="line">        <span class="keyword">while</span> <span class="built_in">len</span>(all_img_urls) &gt;<span class="number">0</span> : </span><br><span class="line">            g_lock.acquire()</span><br><span class="line">            img_url = all_img_urls.pop()</span><br><span class="line">            g_lock.release()</span><br><span class="line">            <span class="keyword">try</span>:</span><br><span class="line">                response = requests.get(img_url , headers = headers )</span><br><span class="line">                response.encoding=<span class="string">&#x27;gb2312&#x27;</span>   <span class="comment">#由于我们调用的页面编码是GB2312，所以需要设置一下编码</span></span><br><span class="line">                title = re.search(<span class="string">&#x27;&lt;title&gt;(.*?) | 妹子图&lt;/title&gt;&#x27;</span>,response.text).group(<span class="number">1</span>)</span><br><span class="line">                all_pic_src = re.findall(<span class="string">&#x27;&lt;img alt=.*?src=&quot;(.*?)&quot; /&gt;&lt;br /&gt;&#x27;</span>,response.text,re.S)</span><br><span class="line">                </span><br><span class="line">                pic_dict = &#123;title:all_pic_src&#125;   <span class="comment">#python字典</span></span><br><span class="line">                <span class="keyword">global</span> pic_links</span><br><span class="line">                g_lock.acquire()</span><br><span class="line">                pic_links.append(pic_dict)    <span class="comment">#字典数组</span></span><br><span class="line">                <span class="built_in">print</span>(title+<span class="string">&quot; 获取成功&quot;</span>)</span><br><span class="line">                g_lock.release()</span><br><span class="line">                </span><br><span class="line">            <span class="keyword">except</span>:</span><br><span class="line">                <span class="keyword">pass</span></span><br><span class="line">            time.sleep(<span class="number">0.5</span>)</span><br></pre></td></tr></table></figure>
<p>代码中比较重要的一些部分，我已经使用注释写好了，大家可以直接参考。大家一定要注意我上面使用了两个正则表达式，分别用来匹配title和图片的url这个title是为了后面创建不同的文件夹使用的，所以大家注意吧。</p>
<pre><code>#开启10个线程去获取链接
for x in range(10):
    ta = Consumer()
    ta.start()
</code></pre><p>运行结果：</p>
<pre><code>[linuxboy@bogon demo]$ python3 down.py
分析http://www.meizitu.com/a/pure_2.html
分析http://www.meizitu.com/a/pure_1.html
[&#39;http://www.meizitu.com/a/5585.html&#39;, ......
&lt;function current_thread at 0x7f7caef851e0&gt; is running 
&lt;function current_thread at 0x7f7caef851e0&gt; is running 
&lt;function current_thread at 0x7f7caef851e0&gt; is running 
&lt;function current_thread at 0x7f7caef851e0&gt; is running 
&lt;function current_thread at 0x7f7caef851e0&gt; is running 
&lt;function current_thread at 0x7f7caef851e0&gt; is running 
&lt;function current_thread at 0x7f7caef851e0&gt; is running 
&lt;function current_thread at 0x7f7caef851e0&gt; is running 
进行到我这里了
&lt;function current_thread at 0x7f7caef851e0&gt; is running 
&lt;function current_thread at 0x7f7caef851e0&gt; is running 
清纯美如画，摄影师的御用麻豆 获取成功
宅男女神叶梓萱近日拍摄一组火爆写真 获取成功
美（bao）胸（ru）女王带来制服诱惑 获取成功
每天睁开眼看到美好的你，便是幸福 获取成功
可爱女孩，愿暖风呵护纯真和执着 获取成功
清纯妹子如一缕阳光温暖这个冬天 获取成功 .....
</code></pre><p>距离成功有进了一大步</p>
<p>接下来就是，我们开篇提到的那个存储图片的操作了，还是同样的步骤，写一个自定义的类</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">class</span> <span class="title class_">DownPic</span>(threading.Thread) :</span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">run</span>(<span class="params">self</span>):</span><br><span class="line">        headers = &#123;</span><br><span class="line">            <span class="string">&#x27;User-Agent&#x27;</span>: <span class="string">&#x27;Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0&#x27;</span>,</span><br><span class="line">            <span class="string">&#x27;HOST&#x27;</span>:<span class="string">&#x27;mm.chinasareview.com&#x27;</span></span><br><span class="line">        </span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">while</span> <span class="literal">True</span>:   <span class="comment">#  这个地方写成死循环，为的是不断监控图片链接数组是否更新</span></span><br><span class="line">            <span class="keyword">global</span> pic_links</span><br><span class="line">            <span class="comment"># 上锁</span></span><br><span class="line">            g_lock.acquire()</span><br><span class="line">            <span class="keyword">if</span> <span class="built_in">len</span>(pic_links) == <span class="number">0</span>:   <span class="comment">#如果没有图片了，就解锁</span></span><br><span class="line">                <span class="comment"># 不管什么情况，都要释放锁</span></span><br><span class="line">                g_lock.release()</span><br><span class="line">                <span class="keyword">continue</span></span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                pic = pic_links.pop()</span><br><span class="line">                g_lock.release()</span><br><span class="line">                <span class="comment"># 遍历字典列表</span></span><br><span class="line">                <span class="keyword">for</span> key,values <span class="keyword">in</span>  pic.items():</span><br><span class="line">                    path=key.rstrip(<span class="string">&quot;\\&quot;</span>)</span><br><span class="line">                    is_exists=os.path.exists(path)</span><br><span class="line">                    <span class="comment"># 判断结果</span></span><br><span class="line">                    <span class="keyword">if</span> <span class="keyword">not</span> is_exists:</span><br><span class="line">                        <span class="comment"># 如果不存在则创建目录</span></span><br><span class="line">                        <span class="comment"># 创建目录操作函数</span></span><br><span class="line">                        os.makedirs(path) </span><br><span class="line">                </span><br><span class="line">                        <span class="built_in">print</span> (path+<span class="string">&#x27;目录创建成功&#x27;</span>)</span><br><span class="line">                        </span><br><span class="line">                    <span class="keyword">else</span>:</span><br><span class="line">                        <span class="comment"># 如果目录存在则不创建，并提示目录已存在</span></span><br><span class="line">                        <span class="built_in">print</span>(path+<span class="string">&#x27; 目录已存在&#x27;</span>) </span><br><span class="line">                    <span class="keyword">for</span> pic <span class="keyword">in</span> values :</span><br><span class="line">                        filename = path+<span class="string">&quot;/&quot;</span>+pic.split(<span class="string">&#x27;/&#x27;</span>)[-<span class="number">1</span>]</span><br><span class="line">                        <span class="keyword">if</span> os.path.exists(filename):</span><br><span class="line">                            <span class="keyword">continue</span></span><br><span class="line">                        <span class="keyword">else</span>:</span><br><span class="line">                            response = requests.get(pic,headers=headers)</span><br><span class="line">                            <span class="keyword">with</span> <span class="built_in">open</span>(filename,<span class="string">&#x27;wb&#x27;</span>) <span class="keyword">as</span> f :</span><br><span class="line">                                f.write(response.content)</span><br><span class="line">                                f.close</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>我们获取图片链接之后，就需要下载了，我上面的代码是首先创建了一个之前获取到title的文件目录，然后在目录里面通过下面的代码,去创建一个文件。</p>
<p>涉及到文件操作，引入一个新的模块<br><code>import os  #目录操作模块</code></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 遍历字典列表</span></span><br><span class="line"><span class="keyword">for</span> key,values <span class="keyword">in</span>  pic.items():</span><br><span class="line">    path=key.rstrip(<span class="string">&quot;\\&quot;</span>)</span><br><span class="line">    is_exists=os.path.exists(path)</span><br><span class="line">    <span class="comment"># 判断结果</span></span><br><span class="line">    <span class="keyword">if</span> <span class="keyword">not</span> is_exists:</span><br><span class="line">        <span class="comment"># 如果不存在则创建目录</span></span><br><span class="line">        <span class="comment"># 创建目录操作函数</span></span><br><span class="line">        os.makedirs(path) </span><br><span class="line"></span><br><span class="line">        <span class="built_in">print</span> (path+<span class="string">&#x27;目录创建成功&#x27;</span>)</span><br><span class="line">        </span><br><span class="line">    <span class="keyword">else</span>:</span><br><span class="line">        <span class="comment"># 如果目录存在则不创建，并提示目录已存在</span></span><br><span class="line">        <span class="built_in">print</span>(path+<span class="string">&#x27; 目录已存在&#x27;</span>) </span><br><span class="line">    <span class="keyword">for</span> pic <span class="keyword">in</span> values :</span><br><span class="line">        filename = path+<span class="string">&quot;/&quot;</span>+pic.split(<span class="string">&#x27;/&#x27;</span>)[-<span class="number">1</span>]</span><br><span class="line">        <span class="keyword">if</span> os.path.exists(filename):</span><br><span class="line">            <span class="keyword">continue</span></span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            response = requests.get(pic,headers=headers)</span><br><span class="line">            <span class="keyword">with</span> <span class="built_in">open</span>(filename,<span class="string">&#x27;wb&#x27;</span>) <span class="keyword">as</span> f :</span><br><span class="line">                f.write(response.content)</span><br><span class="line">                f.close</span><br></pre></td></tr></table></figure>
<p>因为我们的图片链接数组，里面存放是的字典格式，也就是下面这种格式</p>
<pre><code>[&#123;&quot;妹子图1&quot;:[&quot;http://mm.chinasareview.com/wp-content/uploads/2016a/08/24/01.jpg&quot;,&quot;http://mm.chinasareview.com/wp-content/uploads/2016a/08/24/02.jpg&quot;.&quot;http://mm.chinasareview.com/wp-content/uploads/2016a/08/24/03.jpg&quot;]&#125;,&#123;&quot;妹子图2&quot;:[&quot;http://mm.chinasareview.com/wp-content/uploads/2016a/08/24/01.jpg&quot;,&quot;http://mm.chinasareview.com/wp-content/uploads/2016a/08/24/02.jpg&quot;.&quot;http://mm.chinasareview.com/wp-content/uploads/2016a/08/24/03.jpg&quot;]&#125;,&#123;&quot;妹子图3&quot;:[&quot;http://mm.chinasareview.com/wp-content/uploads/2016a/08/24/01.jpg&quot;,&quot;http://mm.chinasareview.com/wp-content/uploads/2016a/08/24/02.jpg&quot;.&quot;http://mm.chinasareview.com/wp-content/uploads/2016a/08/24/03.jpg&quot;]&#125;]
</code></pre><p>需要先循环第一层,获取title，创建目录之后，在循环第二层去下载图片，代码中，我们在修改一下，把异常处理添加上。</p>
<pre><code>try:
    response = requests.get(pic,headers=headers)
    with open(filename,&#39;wb&#39;) as f :
        f.write(response.content)
        f.close
except Exception as e:
    print(e)
    pass
</code></pre><p>然后在主程序中编写代码</p>
<pre><code>#开启10个线程保存图片
for x in range(10):
    down = DownPic()
    down.start()
</code></pre><p>运行结果：</p>
<pre><code>[linuxboy@bogon demo]$ python3 down.py
分析http://www.meizitu.com/a/pure_2.html
分析http://www.meizitu.com/a/pure_1.html
[&#39;http://www.meizitu.com/a/5585.html&#39;, &#39;http://www.meizitu.com/a/5577.html&#39;, &#39;http://www.meizitu.com/a/5576.html&#39;, &#39;http://www.meizitu.com/a/5574.html&#39;, &#39;http://www.meizitu.com/a/5569.html&#39;, .......
&lt;function current_thread at 0x7fa5121f2268&gt; is running 
&lt;function current_thread at 0x7fa5121f2268&gt; is running 
&lt;function current_thread at 0x7fa5121f2268&gt; is running 
进行到我这里了
清纯妹子如一缕阳光温暖这个冬天 获取成功
清纯妹子如一缕阳光温暖这个冬天目录创建成功
可爱女孩，愿暖风呵护纯真和执着 获取成功
可爱女孩，愿暖风呵护纯真和执着目录创建成功
超美，纯纯的你与蓝蓝的天相得益彰 获取成功
超美，纯纯的你与蓝蓝的天相得益彰目录创建成功
美丽冻人，雪地里的跆拳道少女 获取成功
五官精致的美眉，仿佛童话里的公主 获取成功
有自信迷人的笑容，每天都是灿烂的 获取成功
五官精致的美眉，仿佛童话里的公主目录创建成功
有自信迷人的笑容，每天都是灿烂的目录创建成功
清纯美如画，摄影师的御用麻豆 获取成功
</code></pre><p>文件目录下面同时出现</p>
<p><img src="https://s2.ax1x.com/2020/02/04/1BB0Gn.jpg" alt="1BB0Gn.jpg"></p>
<p>点开一个目录</p>
<p><img src="https://s2.ax1x.com/2020/02/04/1BBB2q.jpg" alt="1BBB2q.jpg"></p>
<p>最后我们在代码的头部写上<br><code># -*- coding: UTF-8 -*-</code>以防出现<code>Non-ASCII character &#39;xe5&#39; in file</code>报错问题</p>
<h1 id="完整代码"><a href="#完整代码" class="headerlink" title="完整代码"></a>完整代码</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">import</span> threading  <span class="comment"># 多线程模块</span></span><br><span class="line"><span class="keyword">import</span> re  <span class="comment"># 正则表达式模块</span></span><br><span class="line"><span class="keyword">import</span> time  <span class="comment"># 时间模块</span></span><br><span class="line"><span class="keyword">import</span> os  <span class="comment"># 目录操作模块</span></span><br><span class="line"></span><br><span class="line">all_urls = []  <span class="comment"># 我们拼接好的图片集和列表路径</span></span><br><span class="line">all_img_urls = []  <span class="comment"># 图片列表页面的数组</span></span><br><span class="line">pic_links = []  <span class="comment"># 图片地址列表</span></span><br><span class="line">g_lock = threading.Lock()  <span class="comment"># 初始化一个锁</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Spider</span>():</span><br><span class="line">    <span class="comment"># 构造函数，初始化数据使用</span></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self, target_url, headers</span>):</span><br><span class="line">        self.target_url = target_url</span><br><span class="line">        self.headers = headers</span><br><span class="line"></span><br><span class="line">    <span class="comment"># 获取所有的想要抓取的URL</span></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">getUrls</span>(<span class="params">self, start_page, page_num</span>):</span><br><span class="line">        <span class="comment"># 循环得到URL</span></span><br><span class="line">        <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(start_page, page_num + <span class="number">1</span>):</span><br><span class="line">            url = self.target_url % i</span><br><span class="line">            all_urls.append(url)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 消费者</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Consumer</span>(threading.Thread):</span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">run</span>(<span class="params">self</span>):</span><br><span class="line">        headers = &#123;</span><br><span class="line">            <span class="string">&#x27;User-Agent&#x27;</span>: <span class="string">&#x27;Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0&#x27;</span>,</span><br><span class="line">            <span class="string">&#x27;HOST&#x27;</span>: <span class="string">&#x27;www.meizitu.com&#x27;</span></span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">global</span> all_img_urls</span><br><span class="line">        <span class="built_in">print</span>(<span class="string">&quot;%s is running &quot;</span> % threading.current_thread)</span><br><span class="line">        <span class="keyword">while</span> <span class="built_in">len</span>(all_img_urls) &gt; <span class="number">0</span>:</span><br><span class="line">            g_lock.acquire()</span><br><span class="line">            img_url = all_img_urls.pop()</span><br><span class="line">            g_lock.release()</span><br><span class="line">            <span class="keyword">try</span>:</span><br><span class="line">                response = requests.get(img_url, headers=headers)</span><br><span class="line"></span><br><span class="line">                response.encoding = <span class="string">&#x27;gb2312&#x27;</span></span><br><span class="line">                title = re.search(<span class="string">&#x27;&lt;title&gt;(.*?) | 妹子图&lt;/title&gt;&#x27;</span>, response.text).group(<span class="number">1</span>)</span><br><span class="line">                all_pic_src = re.findall(<span class="string">&#x27;&lt;img alt=.*?src=&quot;(.*?)&quot; /&gt;&lt;br /&gt;&#x27;</span>, response.text, re.S)</span><br><span class="line"></span><br><span class="line">                pic_dict = &#123;title: all_pic_src&#125;</span><br><span class="line">                <span class="keyword">global</span> pic_links</span><br><span class="line">                g_lock.acquire()</span><br><span class="line">                pic_links.append(pic_dict)</span><br><span class="line">                <span class="built_in">print</span>(title + <span class="string">&quot; 获取成功&quot;</span>)</span><br><span class="line">                g_lock.release()</span><br><span class="line"></span><br><span class="line">            <span class="keyword">except</span>:</span><br><span class="line">                <span class="keyword">pass</span></span><br><span class="line">            time.sleep(<span class="number">0.5</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 生产者，负责从每个页面提取图片列表链接</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Producer</span>(threading.Thread):</span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">run</span>(<span class="params">self</span>):</span><br><span class="line">        headers = &#123;</span><br><span class="line">            <span class="string">&#x27;User-Agent&#x27;</span>: <span class="string">&#x27;Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0&#x27;</span>,</span><br><span class="line">            <span class="string">&#x27;HOST&#x27;</span>: <span class="string">&#x27;www.meizitu.com&#x27;</span></span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">global</span> all_urls</span><br><span class="line">        <span class="keyword">while</span> <span class="built_in">len</span>(all_urls) &gt; <span class="number">0</span>:</span><br><span class="line">            g_lock.acquire()  <span class="comment"># 在访问all_urls的时候，需要使用锁机制</span></span><br><span class="line">            page_url = all_urls.pop()  <span class="comment"># 通过pop方法移除最后一个元素，并且返回该值</span></span><br><span class="line"></span><br><span class="line">            g_lock.release()  <span class="comment"># 使用完成之后及时把锁给释放，方便其他线程使用</span></span><br><span class="line">            <span class="keyword">try</span>:</span><br><span class="line">                <span class="built_in">print</span>(<span class="string">&quot;分析&quot;</span> + page_url)</span><br><span class="line">                response = requests.get(page_url, headers=headers, timeout=<span class="number">3</span>)</span><br><span class="line">                all_pic_link = re.findall(<span class="string">&#x27;&lt;a target=\&#x27;_blank\&#x27; href=&quot;(.*?)&quot;&gt;&#x27;</span>, response.text, re.S)</span><br><span class="line">                <span class="keyword">global</span> all_img_urls</span><br><span class="line">                g_lock.acquire()</span><br><span class="line">                all_img_urls += all_pic_link</span><br><span class="line">                <span class="built_in">print</span>(all_img_urls)</span><br><span class="line">                g_lock.release()</span><br><span class="line"></span><br><span class="line">            <span class="keyword">except</span>:</span><br><span class="line">                <span class="keyword">pass</span></span><br><span class="line">            time.sleep(<span class="number">0.5</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">DownPic</span>(threading.Thread):</span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">run</span>(<span class="params">self</span>):</span><br><span class="line">        headers = &#123;</span><br><span class="line">            <span class="string">&#x27;User-Agent&#x27;</span>: <span class="string">&#x27;Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0&#x27;</span>,</span><br><span class="line">            <span class="string">&#x27;HOST&#x27;</span>: <span class="string">&#x27;mm.chinasareview.com&#x27;</span>,</span><br><span class="line">            <span class="string">&#x27;Cookie&#x27;</span>: <span class="string">&#x27;safedog-flow-item=; UM_distinctid=1634d30879a9-06230f37a83b4b8-38694646-1fa400-1634d30879c451; CNZZDATA30056528=cnzz_eid%3D1182386559-1526005899-http%253A%252F%252Fwww.meizitu.com%252F%26ntime%3D1526022264; bdshare_firstime=1526025336868&#x27;</span></span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line">            <span class="keyword">global</span> pic_links</span><br><span class="line">            <span class="comment"># 上锁</span></span><br><span class="line">            g_lock.acquire()</span><br><span class="line">            <span class="keyword">if</span> <span class="built_in">len</span>(pic_links) == <span class="number">0</span>:</span><br><span class="line">                <span class="comment"># 不管什么情况，都要释放锁</span></span><br><span class="line">                g_lock.release()</span><br><span class="line">                <span class="keyword">continue</span></span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                pic = pic_links.pop()</span><br><span class="line">                g_lock.release()</span><br><span class="line">                <span class="comment"># 遍历字典列表</span></span><br><span class="line">                <span class="keyword">for</span> key, values <span class="keyword">in</span> pic.items():</span><br><span class="line">                    path = key.rstrip(<span class="string">&quot;\\&quot;</span>)</span><br><span class="line">                    is_exists = os.path.exists(path)</span><br><span class="line">                    <span class="comment"># 判断结果</span></span><br><span class="line">                    <span class="keyword">if</span> <span class="keyword">not</span> is_exists:</span><br><span class="line">                        <span class="comment"># 如果不存在则创建目录</span></span><br><span class="line">                        <span class="comment"># 创建目录操作函数</span></span><br><span class="line">                        os.makedirs(path)</span><br><span class="line"></span><br><span class="line">                        <span class="built_in">print</span>(path + <span class="string">&#x27;目录创建成功&#x27;</span>)</span><br><span class="line"></span><br><span class="line">                    <span class="keyword">else</span>:</span><br><span class="line">                        <span class="comment"># 如果目录存在则不创建，并提示目录已存在</span></span><br><span class="line">                        <span class="built_in">print</span>(path + <span class="string">&#x27; 目录已存在&#x27;</span>)</span><br><span class="line">                    <span class="keyword">for</span> pic <span class="keyword">in</span> values:</span><br><span class="line">                        filename = path + <span class="string">&quot;/&quot;</span> + pic.split(<span class="string">&#x27;/&#x27;</span>)[-<span class="number">1</span>]</span><br><span class="line">                        <span class="keyword">if</span> os.path.exists(filename):</span><br><span class="line">                            <span class="keyword">continue</span></span><br><span class="line">                        <span class="keyword">else</span>:</span><br><span class="line">                            <span class="keyword">try</span>:</span><br><span class="line">                                response = requests.get(pic, headers=headers)</span><br><span class="line">                                <span class="keyword">with</span> <span class="built_in">open</span>(filename, <span class="string">&#x27;wb&#x27;</span>) <span class="keyword">as</span> f:</span><br><span class="line">                                    f.write(response.content)</span><br><span class="line">                                    f.close</span><br><span class="line">                            <span class="keyword">except</span> Exception <span class="keyword">as</span> e:</span><br><span class="line">                                <span class="built_in">print</span>(e)</span><br><span class="line">                                <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">&quot;__main__&quot;</span>:</span><br><span class="line">    headers = &#123;</span><br><span class="line">        <span class="string">&#x27;User-Agent&#x27;</span>: <span class="string">&#x27;Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0&#x27;</span>,</span><br><span class="line">        <span class="string">&#x27;HOST&#x27;</span>: <span class="string">&#x27;www.meizitu.com&#x27;</span></span><br><span class="line">    &#125;</span><br><span class="line">    target_url = <span class="string">&#x27;http://www.meizitu.com/a/pure_%d.html&#x27;</span>  <span class="comment"># 图片集和列表规则</span></span><br><span class="line"></span><br><span class="line">    spider = Spider(target_url, headers)</span><br><span class="line">    spider.getUrls(<span class="number">1</span>, <span class="number">2</span>)</span><br><span class="line"></span><br><span class="line">    threads = []</span><br><span class="line">    <span class="comment"># 开启两个线程去访问</span></span><br><span class="line">    <span class="keyword">for</span> x <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">2</span>):</span><br><span class="line">        t = Producer()</span><br><span class="line">        t.start()</span><br><span class="line">        threads.append(t)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">for</span> tt <span class="keyword">in</span> threads:</span><br><span class="line">        tt.join()</span><br><span class="line">    <span class="comment"># 开启10个线程去获取链接</span></span><br><span class="line">    <span class="keyword">for</span> x <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>):</span><br><span class="line">        ta = Consumer()</span><br><span class="line">        ta.start()</span><br><span class="line"></span><br><span class="line">    <span class="comment"># 开启5个线程保存图片</span></span><br><span class="line">    <span class="keyword">for</span> x <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>):</span><br><span class="line">        down = DownPic()</span><br><span class="line">        down.start()</span><br><span class="line"></span><br><span class="line">    <span class="built_in">print</span>(<span class="string">&quot;进行到我这里了&quot;</span>)</span><br></pre></td></tr></table></figure>
<h1 id="他解"><a href="#他解" class="headerlink" title="他解"></a>他解</h1><p>我们可以知道，同一个页面需要爬取的内容可能具有多方面的特征，上述的获取所需网页链接的方案因此不唯一。</p>
<p>可以发现图片链接还有以下的特征可以提取：</p>
<p><img src="https://s2.ax1x.com/2020/02/04/1BDqXV.jpg" alt="1BDqXV.jpg">  </p>
<p><img src="https://s2.ax1x.com/2020/02/04/1BDOmT.jpg" alt="1BDOmT.jpg"></p>
<p>因此可以考虑采用以下代码进行爬取：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> threading</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">download_page</span>(<span class="params">url</span>):</span><br><span class="line">    <span class="string">&#x27;&#x27;&#x27;</span></span><br><span class="line"><span class="string">    用于下载页面</span></span><br><span class="line"><span class="string">    &#x27;&#x27;&#x27;</span></span><br><span class="line">    headers = &#123;<span class="string">&quot;User-Agent&quot;</span>: <span class="string">&quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362&quot;</span>&#125;</span><br><span class="line">    r = requests.get(url, headers=headers)</span><br><span class="line">    r.encoding = <span class="string">&#x27;gb2312&#x27;</span></span><br><span class="line">    <span class="keyword">return</span> r.text</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">get_pic_list</span>(<span class="params">html</span>):</span><br><span class="line">    <span class="string">&#x27;&#x27;&#x27;</span></span><br><span class="line"><span class="string">    获取每个页面的套图列表,之后循环调用get_pic函数获取图片</span></span><br><span class="line"><span class="string">    &#x27;&#x27;&#x27;</span></span><br><span class="line">    soup = BeautifulSoup(html, <span class="string">&#x27;html.parser&#x27;</span>)</span><br><span class="line">    pic_list = soup.find_all(<span class="string">&#x27;li&#x27;</span>, class_=<span class="string">&#x27;wp-item&#x27;</span>)</span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> pic_list:</span><br><span class="line">        a_tag = i.find(<span class="string">&#x27;h3&#x27;</span>, class_=<span class="string">&#x27;tit&#x27;</span>).find(<span class="string">&#x27;a&#x27;</span>)</span><br><span class="line">        link = a_tag.get(<span class="string">&#x27;href&#x27;</span>)</span><br><span class="line">        text = a_tag.get_text()</span><br><span class="line">        get_pic(link, text)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">get_pic</span>(<span class="params">link, text</span>):</span><br><span class="line">    <span class="string">&#x27;&#x27;&#x27;</span></span><br><span class="line"><span class="string">    获取当前页面的图片,并保存</span></span><br><span class="line"><span class="string">    &#x27;&#x27;&#x27;</span></span><br><span class="line">    html = download_page(link)  <span class="comment"># 下载界面</span></span><br><span class="line">    soup = BeautifulSoup(html, <span class="string">&#x27;html.parser&#x27;</span>)</span><br><span class="line">    pic_list = soup.find(<span class="string">&#x27;div&#x27;</span>, <span class="built_in">id</span>=<span class="string">&quot;picture&quot;</span>).find_all(<span class="string">&#x27;img&#x27;</span>)  <span class="comment"># 找到界面所有图片</span></span><br><span class="line">    headers = &#123;<span class="string">&quot;User-Agent&quot;</span>: <span class="string">&quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362&quot;</span>&#125;</span><br><span class="line">    create_dir(<span class="string">&#x27;pic/&#123;&#125;&#x27;</span>.<span class="built_in">format</span>(text))</span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> pic_list:</span><br><span class="line">        pic_link = i.get(<span class="string">&#x27;src&#x27;</span>)  <span class="comment"># 拿到图片的具体 url</span></span><br><span class="line">        r = requests.get(pic_link, headers=headers)  <span class="comment"># 下载图片，之后保存到文件</span></span><br><span class="line">        <span class="keyword">with</span> <span class="built_in">open</span>(<span class="string">&#x27;pic/&#123;&#125;/&#123;&#125;&#x27;</span>.<span class="built_in">format</span>(text, pic_link.split(<span class="string">&#x27;/&#x27;</span>)[-<span class="number">1</span>]), <span class="string">&#x27;wb&#x27;</span>) <span class="keyword">as</span> f:</span><br><span class="line">            f.write(r.content)</span><br><span class="line">            time.sleep(<span class="number">3</span>)   <span class="comment"># 休息一下，不要给网站太大压力，避免被封</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">create_dir</span>(<span class="params">name</span>):</span><br><span class="line">    <span class="keyword">if</span> <span class="keyword">not</span> os.path.exists(name):</span><br><span class="line">        os.makedirs(name)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">execute</span>(<span class="params">url</span>):</span><br><span class="line">    page_html = download_page(url)</span><br><span class="line">    get_pic_list(page_html)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">main</span>():</span><br><span class="line">    create_dir(<span class="string">&#x27;pic&#x27;</span>)</span><br><span class="line">    queue = [i <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, <span class="number">72</span>)]   <span class="comment"># 构造 url 链接 页码。</span></span><br><span class="line">    threads = []</span><br><span class="line">    <span class="keyword">while</span> <span class="built_in">len</span>(queue) &gt; <span class="number">0</span>:</span><br><span class="line">        <span class="keyword">for</span> thread <span class="keyword">in</span> threads:</span><br><span class="line">            <span class="keyword">if</span> <span class="keyword">not</span> thread.is_alive():</span><br><span class="line">                threads.remove(thread)</span><br><span class="line">        <span class="keyword">while</span> <span class="built_in">len</span>(threads) &lt; <span class="number">5</span> <span class="keyword">and</span> <span class="built_in">len</span>(queue) &gt; <span class="number">0</span>:   <span class="comment"># 最大线程数设置为 5</span></span><br><span class="line">            cur_page = queue.pop(<span class="number">0</span>)</span><br><span class="line">            url = <span class="string">&#x27;http://meizitu.com/a/more_&#123;&#125;.html&#x27;</span>.<span class="built_in">format</span>(cur_page)</span><br><span class="line">            thread = threading.Thread(target=execute, args=(url,))</span><br><span class="line">            thread.setDaemon(<span class="literal">True</span>)</span><br><span class="line">            thread.start()</span><br><span class="line">            <span class="built_in">print</span>(<span class="string">&#x27;&#123;&#125;正在下载&#123;&#125;页&#x27;</span>.<span class="built_in">format</span>(threading.current_thread().name, cur_page))</span><br><span class="line">            threads.append(thread)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">&#x27;__main__&#x27;</span>:</span><br><span class="line">    main()</span><br></pre></td></tr></table></figure>
 
      <!-- reward -->
      
      <div id="reword-out">
        <div id="reward-btn">
          打赏
        </div>
      </div>
      
    </div>
    

    <!-- copyright -->
    
    <div class="declare">
      <ul class="post-copyright">
        <li>
          <i class="ri-copyright-line"></i>
          <strong>版权声明： </strong>
          
          本博客所有文章除特别声明外，著作权归作者所有。转载请注明出处！
          
        </li>
      </ul>
    </div>
    
    <footer class="article-footer">
       
<div class="share-btn">
      <span class="share-sns share-outer">
        <i class="ri-share-forward-line"></i>
        分享
      </span>
      <div class="share-wrap">
        <i class="arrow"></i>
        <div class="share-icons">
          
          <a class="weibo share-sns" href="javascript:;" data-type="weibo">
            <i class="ri-weibo-fill"></i>
          </a>
          <a class="weixin share-sns wxFab" href="javascript:;" data-type="weixin">
            <i class="ri-wechat-fill"></i>
          </a>
          <a class="qq share-sns" href="javascript:;" data-type="qq">
            <i class="ri-qq-fill"></i>
          </a>
          <a class="douban share-sns" href="javascript:;" data-type="douban">
            <i class="ri-douban-line"></i>
          </a>
          <!-- <a class="qzone share-sns" href="javascript:;" data-type="qzone">
            <i class="icon icon-qzone"></i>
          </a> -->
          
          <a class="facebook share-sns" href="javascript:;" data-type="facebook">
            <i class="ri-facebook-circle-fill"></i>
          </a>
          <a class="twitter share-sns" href="javascript:;" data-type="twitter">
            <i class="ri-twitter-fill"></i>
          </a>
          <a class="google share-sns" href="javascript:;" data-type="google">
            <i class="ri-google-fill"></i>
          </a>
        </div>
      </div>
</div>

<div class="wx-share-modal">
    <a class="modal-close" href="javascript:;"><i class="ri-close-circle-line"></i></a>
    <p>扫一扫，分享到微信</p>
    <div class="wx-qrcode">
      <img src="//api.qrserver.com/v1/create-qr-code/?size=150x150&data=https://blog.shimmerjordan.eu.org/2020/02/02/Python-crawlers-beginning/" alt="微信分享二维码">
    </div>
</div>

<div id="share-mask"></div>  
  <ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/Python/" rel="tag">Python</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/%E5%A4%9A%E7%BA%BF%E7%A8%8B/" rel="tag">多线程</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/%E7%88%AC%E8%99%AB/" rel="tag">爬虫</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/%E8%A7%82%E5%AF%9F%E8%80%85%E6%A8%A1%E5%BC%8F/" rel="tag">观察者模式</a></li></ul>

    </footer>
  </div>

   
  <nav class="article-nav">
    
      <a href="/2020/02/13/intro-SPFA/" class="article-nav-link">
        <strong class="article-nav-caption">上一篇</strong>
        <div class="article-nav-title">
          
            SPFA算法详解
          
        </div>
      </a>
    
    
      <a href="/2020/01/29/DSTServerConstruction/" class="article-nav-link">
        <strong class="article-nav-caption">下一篇</strong>
        <div class="article-nav-title">饥荒云服搭搭建</div>
      </a>
    
  </nav>

   
<!-- valine评论 -->
<div id="vcomments-box">
  <div id="vcomments"></div>
</div>
<script src="//cdn1.lncld.net/static/js/3.0.4/av-min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/valine@1.4.14/dist/Valine.min.js"></script>
<script>
  new Valine({
    el: "#vcomments",
    app_id: "StYfTMDp78X0EltFR16ve2q5-gzGzoHsz",
    app_key: "G4RPxRpXG6RwdfpnJefOSnyy",
    path: window.location.pathname,
    avatar: "wavatar",
    placeholder: "ヾﾉ≧∀≦)o来啊，快活啊!",
    recordIP: true,
  });
  const infoEle = document.querySelector("#vcomments .info");
  if (infoEle && infoEle.childNodes && infoEle.childNodes.length > 0) {
    infoEle.childNodes.forEach(function (item) {
      item.parentNode.removeChild(item);
    });
  }
</script>
<style>
  #vcomments-box {
    padding: 5px 30px;
  }

  @media screen and (max-width: 800px) {
    #vcomments-box {
      padding: 5px 0px;
    }
  }

  #vcomments-box #vcomments {
    background-color: #fff;
  }

  .v .vlist .vcard .vh {
    padding-right: 20px;
  }

  .v .vlist .vcard {
    padding-left: 10px;
  }
</style>

 
   
   
<!-- minivaline评论 -->
<div id="mvcomments-box">
  <div id="mvcomments"></div>
</div>
<script src="https://cdn.jsdelivr.net/npm/minivaline@latest"></script>
<script>
    new MiniValine(Object.assign({"enable":true,"mode":"DesertsP","placeholder":"Write a Comment","math":true,"md":true,"enableQQ":true,"NoRecordIP":false,"visitor":true,"maxNest":6,"pageSize":6,"adminEmailMd5":"de8a7aa53d07e6b6bceb45c64027763d","tagMeta":["管理员","小伙伴","访客"],"master":["de8a7aa53d07e6b6bceb45c64027763d"],"friends":["b5bd5d836c7a0091aa8473e79ed4c25e","adb7d1cd192658a55c0ad22a3309cecf","3ce1e6c77b4910f1871106cb30dc62b0","cfce8dc43725cc14ffcd9fb4892d5bfc"],"lang":null,"emoticonUrl":["https://cdn.jsdelivr.net/npm/alus@latest","https://cdn.jsdelivr.net/gh/MiniValine/qq@latest","https://cdn.jsdelivr.net/gh/MiniValine/Bilibilis@latest","https://cdn.jsdelivr.net/gh/MiniValine/tieba@latest","https://cdn.jsdelivr.net/gh/MiniValine/twemoji@latest","https://cdn.jsdelivr.net/gh/MiniValine/weibo@latest"]}, {
	  el: '#mvcomments',
    }));
  const infoEle = document.querySelector('#mvcomments .info');
  if (infoEle && infoEle.childNodes && infoEle.childNodes.length > 0) {
      infoEle.childNodes.forEach(function (item) {
          item.parentNode.removeChild(item);
      });
  }
</script>
<style>
	#mvcomments-box {
		padding: 5px 30px;
	}
	@media screen and (max-width: 800px) {
		#mvcomments-box {
		  padding: 5px 0px;
		}
	}
	.darkmode .MiniValine *{
		color: #f1f1f1!important;
	}
	.darkmode .commentTrigger{
		background-color: #403e3e !important;
	  }
	.darkmode .MiniValine .vpage .more{
		background: #21232F
	}
	.darkmode img{
		filter: brightness(30%)
	}
	.darkmode .MiniValine .vlist .vcard .vcomment-body .text-wrapper .vcomment.expand:before{
		background: linear-gradient(180deg, rgba(246,246,246,0), rgba(0,0,0,0.9))
	}
	.darkmode .MiniValine .vlist .vcard .vcomment-body .text-wrapper .vcomment.expand:after{
		background: rgba(0,0,0,0.9)
	}
	.darkmode .MiniValine .vlist .vcard .vcomment-body .text-wrapper .vcomment pre{
		background: #282c34
		border: 1px solid #282c34
	}
	.darkmode .MiniValine .vinputs-area .textarea-wrapper textarea{
		color: #000;
	}
	.darkmode .MiniValine .vinputs-area .auth-section .input-wrapper input{
		color: #000;
	}
	.darkmode .MiniValine .vinputs-area .vextra-area .vsmile-icons{
		background: transparent;
	}
	.darkmode .MiniValine .vinputs-wrap{
		border-color: #b2b2b5;
	}
	.darkmode .MiniValine .vinputs-wrap:hover{
		border: 1px dashed #2196f3;
	}
	.darkmode .MiniValine .vinputs-area .auth-section .input-wrapper{
		border-bottom: 1px dashed #b2b2b5;
	}
	.darkmode .MiniValine .vinputs-area .auth-section .input-wrapper:hover{
		border-bottom: 1px dashed #2196f3;
	}
	.darkmode .MiniValine .vbtn{
		background-color: transparent!important;
	}
	.darkmode .MiniValine .vbtn:hover{
		border: 1px dashed #2196f3;
	}
</style>

    
</article>

</section>
      <footer class="footer">
  <div class="outer">
    <ul>
      <li>
        Copyrights &copy;
        2019-2024
        <i class="ri-heart-fill heart_icon"></i> 鞠桥丹-QIAODAN JU
      </li>
    </ul>
    <ul>
      <li>
        
        
        
        由 <a href="https://hexo.io" target="_blank">Hexo</a> 强力驱动
        <span class="division">|</span>
        主题 - <a href="https://github.com/Shen-Yu/hexo-theme-ayer" target="_blank">Ayer</a>
        
      </li>
    </ul>
    <ul>
      <li>
        
        
        <span>
  <span><i class="ri-user-3-fill"></i>访问人数:<span id="busuanzi_value_site_uv"></span></s>
  <span class="division">|</span>
  <span><i class="ri-eye-fill"></i>浏览次数:<span id="busuanzi_value_page_pv"></span></span>
</span>
        
      </li>
    </ul>
    <ul>
      
    </ul>
    <ul>
      
    </ul>
    <ul>
      <li>
        <!-- cnzz统计 -->
        
        <script type="text/javascript" src='https://s4.cnzz.com/z_stat.php?id=1279035150&amp;web_id=1279035150'></script>
        
      </li>
    </ul>
  </div>
</footer>
      <div class="float_btns">
        <div class="totop" id="totop">
  <i class="ri-arrow-up-line"></i>
</div>

<div class="todark" id="todark">
  <i class="ri-moon-line"></i>
</div>

      </div>
    </main>
    <aside class="sidebar on">
      <button class="navbar-toggle"></button>
<nav class="navbar">
  
  <div class="logo">
    <a href="/"><img src="/images/ayer-side.svg" alt="丛烨-shimmerjordan"></a>
  </div>
  
  <ul class="nav nav-main">
    
    <li class="nav-item">
      <a class="nav-item-link" href="/">Home</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/archives">Catalogue</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/tags">Tags</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/tags/%E9%9A%8F%E7%AC%94/">Essay</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/categories">Archives</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/friends">Friends</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/2020/01/18/about">About</a>
    </li>
    
  </ul>
</nav>
<nav class="navbar navbar-bottom">
  <ul class="nav">
    <li class="nav-item">
      
      <a class="nav-item-link nav-item-search"  title="搜索">
        <i class="ri-search-line"></i>
      </a>
      
      
      <a class="nav-item-link" target="_blank" href="/atom.xml" title="RSS Feed">
        <i class="ri-rss-line"></i>
      </a>
      
    </li>
  </ul>
</nav>
<div class="search-form-wrap">
  <div class="local-search local-search-plugin">
  <input type="search" id="local-search-input" class="local-search-input" placeholder="Search...">
  <div id="local-search-result" class="local-search-result"></div>
</div>
</div>
    </aside>
    <script>
      if (window.matchMedia("(max-width: 768px)").matches) {
        document.querySelector('.content').classList.remove('on');
        document.querySelector('.sidebar').classList.remove('on');
      }
    </script>
    <div id="mask"></div>

<!-- #reward -->
<div id="reward">
  <span class="close"><i class="ri-close-line"></i></span>
  <p class="reward-p"><i class="ri-cup-line"></i>请我喝杯蓝莓汁吧~</p>
  <div class="reward-box">
    
    <div class="reward-item">
      <img class="reward-img" src="/images/alipay.jpg">
      <span class="reward-type">支付宝</span>
    </div>
    
    
    <div class="reward-item">
      <img class="reward-img" src="/images/wechat.jpg">
      <span class="reward-type">微信</span>
    </div>
    
  </div>
</div>
    
<script src="/js/jquery-2.0.3.min.js"></script>


<script src="/js/lazyload.min.js"></script>

<!-- Tocbot -->


<script src="/js/tocbot.min.js"></script>

<script>
  tocbot.init({
    tocSelector: '.tocbot',
    contentSelector: '.article-entry',
    headingSelector: 'h1, h2, h3, h4, h5, h6',
    hasInnerContainers: true,
    scrollSmooth: true,
    scrollContainer: 'main',
    positionFixedSelector: '.tocbot',
    positionFixedClass: 'is-position-fixed',
    fixedSidebarOffset: 'auto'
  });
</script>

<script src="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.css">
<script src="https://cdn.jsdelivr.net/npm/justifiedGallery@3.7.0/dist/js/jquery.justifiedGallery.min.js"></script>

<script src="/dist/main.js"></script>

<!-- ImageViewer -->

<!-- Root element of PhotoSwipe. Must have class pswp. -->
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">

    <!-- Background of PhotoSwipe. 
         It's a separate element as animating opacity is faster than rgba(). -->
    <div class="pswp__bg"></div>

    <!-- Slides wrapper with overflow:hidden. -->
    <div class="pswp__scroll-wrap">

        <!-- Container that holds slides. 
            PhotoSwipe keeps only 3 of them in the DOM to save memory.
            Don't modify these 3 pswp__item elements, data is added later on. -->
        <div class="pswp__container">
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
        </div>

        <!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
        <div class="pswp__ui pswp__ui--hidden">

            <div class="pswp__top-bar">

                <!--  Controls are self-explanatory. Order can be changed. -->

                <div class="pswp__counter"></div>

                <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>

                <button class="pswp__button pswp__button--share" style="display:none" title="Share"></button>

                <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>

                <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>

                <!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
                <!-- element will get class pswp__preloader--active when preloader is running -->
                <div class="pswp__preloader">
                    <div class="pswp__preloader__icn">
                        <div class="pswp__preloader__cut">
                            <div class="pswp__preloader__donut"></div>
                        </div>
                    </div>
                </div>
            </div>

            <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
                <div class="pswp__share-tooltip"></div>
            </div>

            <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
            </button>

            <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
            </button>

            <div class="pswp__caption">
                <div class="pswp__caption__center"></div>
            </div>

        </div>

    </div>

</div>

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/default-skin/default-skin.min.css">
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe-ui-default.min.js"></script>

<script>
    function viewer_init() {
        let pswpElement = document.querySelectorAll('.pswp')[0];
        let $imgArr = document.querySelectorAll(('.article-entry img:not(.reward-img)'))

        $imgArr.forEach(($em, i) => {
            $em.onclick = () => {
                // slider展开状态
                // todo: 这样不好，后面改成状态
                if (document.querySelector('.left-col.show')) return
                let items = []
                $imgArr.forEach(($em2, i2) => {
                    let img = $em2.getAttribute('data-idx', i2)
                    let src = $em2.getAttribute('data-target') || $em2.getAttribute('src')
                    let title = $em2.getAttribute('alt')
                    // 获得原图尺寸
                    const image = new Image()
                    image.src = src
                    items.push({
                        src: src,
                        w: image.width || $em2.width,
                        h: image.height || $em2.height,
                        title: title
                    })
                })
                var gallery = new PhotoSwipe(pswpElement, PhotoSwipeUI_Default, items, {
                    index: parseInt(i)
                });
                gallery.init()
            }
        })
    }
    viewer_init()
</script>

<!-- MathJax -->

<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
      tex2jax: {
          inlineMath: [ ['$','$'], ["\\(","\\)"]  ],
          processEscapes: true,
          skipTags: ['script', 'noscript', 'style', 'textarea', 'pre', 'code']
      }
  });

  MathJax.Hub.Queue(function() {
      var all = MathJax.Hub.getAllJax(), i;
      for(i=0; i < all.length; i += 1) {
          all[i].SourceElement().parentNode.className += ' has-jax';
      }
  });
</script>

<script src="https://cdn.jsdelivr.net/npm/mathjax@2.7.6/unpacked/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script>
  var ayerConfig = {
    mathjax: true
  }
</script>

<!-- Katex -->

<!-- busuanzi  -->


<script src="/js/busuanzi-2.3.pure.min.js"></script>


<!-- ClickLove -->


<script src="/js/clickLove.js"></script>


<!-- ClickBoom1 -->

<!-- ClickBoom2 -->

<!-- CodeCopy -->


<link rel="stylesheet" href="/css/clipboard.css">

<script src="https://cdn.jsdelivr.net/npm/clipboard@2/dist/clipboard.min.js"></script>
<script>
  function wait(callback, seconds) {
    var timelag = null;
    timelag = window.setTimeout(callback, seconds);
  }
  !function (e, t, a) {
    var initCopyCode = function(){
      var copyHtml = '';
      copyHtml += '<button class="btn-copy" data-clipboard-snippet="">';
      copyHtml += '<i class="ri-file-copy-2-line"></i><span>COPY</span>';
      copyHtml += '</button>';
      $(".highlight .code pre").before(copyHtml);
      $(".article pre code").before(copyHtml);
      var clipboard = new ClipboardJS('.btn-copy', {
        target: function(trigger) {
          return trigger.nextElementSibling;
        }
      });
      clipboard.on('success', function(e) {
        let $btn = $(e.trigger);
        $btn.addClass('copied');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-checkbox-circle-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPIED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-checkbox-circle-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
      clipboard.on('error', function(e) {
        e.clearSelection();
        let $btn = $(e.trigger);
        $btn.addClass('copy-failed');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-time-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPY FAILED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-time-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
    }
    initCopyCode();
  }(window, document);
</script>


<!-- CanvasBackground -->


<script src="/js/dz.js"></script>



    
  </div>
</body>

</html>